Web site analyzers, which are often referred to as indexers, spiders, bots, or crawlers, for example, navigate web sites and collect information regarding their structure or content. These analyzers have many uses including identifying security threats in a web site, evaluating web pages associated with a web site for implementation quality, and producing a list or sitemap of web pages of a web site that should be indexed by search engines, for example. In some instances, analyzers can be used to generate client-facing content, such as a list of available products with current offers in a retailer web site or a static version of a retailer web site catalog that can be used as a catalog navigation menu by a mobile application associated with the retailer or web site, for example.
However, current web site analyzers are limited to inspecting static web page documents associated with web sites. As a result, current web site analyzers do not effectively execute some web pages, including those web pages that include client-side JavaScript code. More specifically, current web site analyzers are unable to extract information from web pages that is hidden inside the client-side JavaScript code. In one particular example, a catalog navigation menu may be generated, when a web page is executed client-side, by downloading and processing a JavaScript Object Notation (JSON) resource, which would not be accessible or executable by current web site analyzers. Therefore, current analyzers have limited functionality and visibility into certain web sites resulting in relatively inaccurate or incomplete results that have limited utility.